Add metastore API spec #7

vikrantpuppala · 2024-09-12T09:24:21Z

PR Checklist

A description of the changes is added to the description of this PR.
If there is a related issue, make sure it is linked to this PR.
If you've fixed a bug or added code that should be tested, add tests!
If you've added or modified a feature, documentation in docs is updated

Description of changes

@creechy

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Sister of PR unitycatalog#379 cc: @creechy

**Description of changes**  Use the Delta 3.2.1 RC so that these ignored Delta tests can be enabled. Also fixes the gs/abfs credential support in the Delta tests, by disabling the file system cache, the same as we did for s3. --------- Signed-off-by: Wenchen Fan <cloud0fan@gmail.com>

**Description of changes**  UC managed tables should use the managed locations from the UC server. We should not leave it to the underlying `DeltaCatalog` which will generate the default table location using the Spark warehouse path. --------- Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

…tycatalog#440) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [X] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** By default, uc-server runs on ports 8080 (transcode) and 8081. Without --server argument, UC CLI/UI only connects to the transcode port 8080 by default. This change is to expose both port 8080/8081 in start-uc-server-docker so that * Users following documents (default: 8080) won't fail to connect the server running in docker. * CLI/UI won't need to specify --server argument. ### Before the Fix: ``` derek@dev-linux unitycatalog (docker_expose) $ docker/bin/start-uc-server-docker ❤️ Starting unitycatalog:0.2.0-SNAPSHOT. 💡 Use Ctrl+C to stop and remove the container. ... derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list --server http://localhost:8081 ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list Exception in thread "main" java.lang.RuntimeException: io.unitycatalog.client.ApiException: java.net.ConnectException at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:165) Caused by: io.unitycatalog.client.ApiException: java.net.ConnectException ... ``` ### After the Fix. ``` derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ ``` --------- Co-authored-by: yaoderek <yaoderek@outlook.com>

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  The API yaml file was updated but the docs weren't, just bringing them back in sync

**PR Checklist** - [x ] A description of the changes is added to the description of this PR. - [x ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Allows users to login via google auth once auth configuration is in place and users have been added to users table. <img width="1064" alt="Screenshot 2024-09-11 at 12 37 00 PM" src="https://github.com/user-attachments/assets/7f5db283-2604-4e6a-a179-f17d6c5537d1"> <img width="323" alt="Screenshot 2024-09-11 at 12 38 38 PM" src="https://github.com/user-attachments/assets/1824e68a-6a76-4f11-87f4-d49522a54a16"> Closes unitycatalog#430

…ocs (unitycatalog#469) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Added owner, createdBy and updatedBy to all securable info objects and database tables Regenerated openAPI docs Updated the id names for model versions and registered models

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Format the test file using javafmtAll

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Enable identity tracking for all create and update paths for securables in UC OSS

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Specify release settings for spark connector

This PR is to test parquet table creation by create table or create table select as

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix a few issues in the 0.2.0 release process. - explicitly specify release settings on a few subprojects - fix some cross-building issues - move `controlApi` dependency to CLI project, it doesn't need to be published / marked as a spark dependency

**PR Checklist** - [x ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Users can browse models in the UI, view details of models, including versions. Version details page displays version data (created_by and updated_by coming soon with access control changes). <img width="1299" alt="Screenshot 2024-09-10 at 12 15 43 PM" src="https://github.com/user-attachments/assets/1591390b-d986-4ab3-8d2e-9c1bf66fcb97"> <img width="1392" alt="Screenshot 2024-09-10 at 12 16 02 PM" src="https://github.com/user-attachments/assets/2982ed79-91d2-49a9-8391-f31861e3a9fe"> <img width="1304" alt="Screenshot 2024-09-10 at 12 16 15 PM" src="https://github.com/user-attachments/assets/9dc8340e-8bc5-4e9f-80a9-242c3808e26e"> <img width="397" alt="Screenshot 2024-09-10 at 12 36 31 PM" src="https://github.com/user-attachments/assets/6c69a6a9-7864-479b-9078-06ab63c5f45b">

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Added temporary path credentials openapi spec and implementation

…log#445) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Invalid entity type leads to Null pointer Exception in output. This change makes sure that entity type is validated first before further checks. (entering a invalid entity type into bin/uc) Before the fix: ``` derek@dev-linux unitycatalog (main) $ bin/uc sdfjio Please provide an operation. Please provide a valid sub-command for sdfjio. Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.util.Map.keySet()" because "entityOptions" is null at io.unitycatalog.cli.utils.CliUtils.printEntityHelp(CliUtils.java:525) at io.unitycatalog.cli.UnityCatalogCli.validateCommand(UnityCatalogCli.java:177) at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:105) ``` After the fix: ``` derek@dev-linux unitycatalog (fix_npe) $ bin/uc sdfjio Invalid entity provided: sdfjio Usage: bin/uc <entity> <operation> [options] Entities: schema, volume, model_version, auth, catalog, function, registered_model, user, table By default, the client will connect to UC running locally at http://localhost:8080 To connect to specific UC server, use --server https://<host>:<port> Currently, auth using bearer token is supported. Please specify the token via --auth_token <PAT Token> For detailed help on entity specific operations, use bin/uc <entity> --help ``` Co-authored-by: yaoderek <yaoderek@outlook.com> Co-authored-by: Denny Lee <denny.g.lee@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Removed the warning message. See [discussion](unitycatalog#349 (comment)). - Add a title, an intro and an [image](https://github.com/unitycatalog/unitycatalog#interact-with-the-unity-catalog-ui). - As an idea, **I add a broken link** - [UI Documentation](https://github.com/unitycatalog/unitycatalog/tree/main/docs/ui). - Update the Get started section. - Add a References section to notate the merged. - Commented out Authenticate and Login section. Please let me know what you think.  --------- Signed-off-by: Juan Diaz <juanchodis@hotmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** We don't want the dependency resolvers to pull in an extra version of spark

**PR Checklist** - [ x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Add the `owner` `created_by` and `updated_by` fields to all other entities (models already has them from previous PR). If the fields are populated, they will display in the details sidebar of each entity. Also, removing non-functional "add/edit description" button from version details page (missed this straggler in previous PR).

…UC OSS (unitycatalog#448) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Added a script that acts as an integration test.

…nitycatalog#427) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fixes unitycatalog#425 Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

This PR adds a documentation page that shows users how to work with ORC tables. Will close unitycatalog#206 --------- Signed-off-by: Avril Aysha <68642378+avriiil@users.noreply.github.com>

This PR adds a documentation page for managed vs external tables. Will close unitycatalog#221

**Description of changes**  If Delta Lake is not included in the class path, UC should still work and support file sources. --------- Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

**Description of changes** Updates the documentation around integration with Daft so that the guide can be followed without running into errors. - `daft.read_delta_lake` does not exist, the right API is `daft.read_deltalake` (see the related Daft docs [here](https://www.getdaft.io/projects/docs/en/latest/api_docs/doc_gen/io_functions/daft.read_deltalake.html#daft.read_deltalake)) - Daft must be installed including extras `unity` and `deltalake`, otherwise `daft.read_deltalake` will raise `ModuleNotFoundError` Signed-off-by: Péter Gyarmati <dev.petergy@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Initial version of model documentation. Quickstart manually verified on my local machine. --------- Co-authored-by: Matthew Powers <matthewkevinpowers@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Name validation may fail when creating a new catalog, but it always shows a general error message which can be misleading. Add a detailed message to the error hint to help users fix their catalog name inputs. Before: ![image](https://github.com/user-attachments/assets/d82a8888-0094-4af1-aedf-a565aada890f) After: <img width="1229" alt="image" src="https://github.com/user-attachments/assets/4dc8bdbc-bec0-48a4-9134-035d68e54b87">

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** As the title says

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** This feature builds upon the authentication functionality to add access controls to UC server endpoints. - Access control is optionally enabled through server server configuration (`server.authorization=enable`) - Resource endpoints all have access control rules that match access rules for Databricks Unity Catalog. - An access control is defined by a principal, a resource and a privilege. - Principals are currently only user entities. - Access controls are hierarchically applied - A privilege assigned to a resource for a principal is active for all the children. - A command-line interface is available to add, update, and delete authorizations. - The root of the hierarchy is a catalog. - There is a new concept of a metastore, which is primarily a resource to assign server level privileges with. The initial set of privileges defined are - `CREATE CATALOG` - allows the principal to create catalogs - `USE CATALOG` - allows the principal to access/use a catalog - `CREATE SCHEMA` - allows the principal to create schemas within a catalog - `USE SCHEMA` - allows the principal to access/use the schema and child tables - `CREATE_TABLE` - allows the principal to create tables in the schema - `SELECT` - allows the principal to run queries against table(s) - `CREATE_FUNCTION` - allows principal to create functions in the schema - `EXECUTE` - allows the principal to execute function(s) - `CREATE_VOLUME` - allows principal to create volumes in the schema - `READ VOLUME` - allows the principal to access volumes within the catalog - `CREATE MODEL` - allows the principal to create models within a schema The rules are generally more complex than summarized above, check Databricks documentation or the code for more details. The CLI interface provides a new `permission` command. Examples ``` bin/uc permission create --securable_type catalog \ --name mycatalog --principal user@myorg.com --privilege "CREATE CATALOG` bin/uc permission get --securable_type catalog --name unity binuc permission delete --securable_type table \ --name mycatalog.myschema.table --principal user@myorg.com --privilege SELECT ``` The framework attempts to separate the concern of access control from the rest of the request handling and processing through annotation-based configuration, via `@AuthorizeExpression`, `@AuthorizeKey` and `@AuthorizeKeys` added to each service entry point. All requests are routed through the `UnityAccessDecorator` which decodes the request, parameters and authorization configuration to evaluate whether access should be allowed or denied. --------- Co-authored-by: Vikrant Puppala <vikrant.puppala@databricks.com> Co-authored-by: Vikrant Puppala <vikrantpuppala@gmail.com> Co-authored-by: Xiang Xu <xiang.xu@databricks.com>

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Database index names apparently need to be unique. The same name `IDX_NAME` was used in multiple locations. Related Issue unitycatalog#493 Co-authored-by: Vikrant Puppala <vikrant.puppala@databricks.com>

…log#509) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Simple wording change for CLI login process.

**PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Minor formatting pages to the docs/quickstart including the MLflow section.

) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** This is a follow on to the overall access control feature to add access control to the credential vending endpoints. The access control rules for credential vending follows the same rules as getting the entity. This resolve Issue unitycatalog#501

…ycatalog#518) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated Remove `PATH_REFRESH` as it is an unused path operation.  Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

**PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed various formatting and configuration settings for the docs site. - Added Spark integration - Added Model integration - Includes coded tabs, admonitions, linked tabs, and other docs improvements.  --------- Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Co-authored-by: Kris Concepcion <84737625+kriscon-db@users.noreply.github.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix javadoc generation issue e.g. `unknown tag: AuthorizeKey`

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Including auth and UI docs. Slight restructuring of docs hierarchy. Clean up missing link, add cross linking, etc.  --------- Co-authored-by: Jamie Knight <jamie.knight@databricks.com> Co-authored-by: Shawn Chen <ychshawn@gmail.com> Co-authored-by: Mocker <1179360+creechy@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Ability to delete a model version from the UI. <img width="425" alt="Screenshot 2024-09-26 at 2 29 57 PM" src="https://github.com/user-attachments/assets/fb27e047-f642-4221-8ef2-3b5ea96091a5"> <img width="522" alt="Screenshot 2024-09-26 at 2 29 51 PM" src="https://github.com/user-attachments/assets/d3b83207-3b21-44c5-8089-04d5370c3118"> <img width="226" alt="Screenshot 2024-09-26 at 2 29 41 PM" src="https://github.com/user-attachments/assets/b4726f0e-b5ad-4ac5-a1b6-77c9a3aba815">

**PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed image loc in UI doc - Added managing users to CLI doc

**PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Clarification of Identity providers supported  Signed-off-by: Denny Lee <denny.g.lee@gmail.com>

**PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Only run `build/sbt generate` to update the docs without any other changes Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762394 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j9oACgkQMF6GPKJC 6IkmsRAAkfo2chEs+B32KbjXUAM+bPmA1j1nnwFc+s3e3RawSKA1g5YPpsysJHJT QjqKcE49WjJ84IHXiFAHsfmsI6RNKkHnwM/0jHxQu7y6zrewXvXNDVfmN6FzcnXH S9vsSAxnqaveS1SAUPA6HpWPQIcRdjFdIHuNB5/lYxOvqUUMb9XsSYLipF6scauN Oh3xS5P3C5EPx+kMKqiCHv/fhZDJZjZjKU4/yDdnFLvjMQgq/WKuGIDeGD6fjbjK U80OO3mRWcTEhCEwUwB62PkzC7F9ckaXGEoq9Hadi33ao3foda+IjM1231Zo7plh lkrYVk0fD37tndfCNKr7OOtaZHFa3lb66Exev3exK5n9e/U97Hlu7JYmw6TsRjTN Ap5g7iOZ+YEuFp4H+UoWZNtW0W9lUIcHLdsZcIFh8fKa0kVtkY1ShsyazFQgNBmP XMO/+pmLEYAgQYq58LjbFWAp63H1oCGBqAZvy+To5p+bbz5fLddbX/dxIzMYupgh PEpcTVBrOunaGm5OE76zS9Z26QNZ6NnmKoHjZhlUYpNPo8swPgx1Nhmdnt4wTl3M omEb1ino4wQYzOVqa0Bs4iPpA/mcZ3EwhuoeuiVVquZUCaeWrOCsrpygMhSM1UvF 9+bwAjfQzKJ7r192HZTH1vO9mGGHgVNNY5jdEjS3yhdjrjWcG7c= =vfl9 -----END PGP SIGNATURE----- parent f58216efe8b13e487e95a058cd20d84690125755 author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762374 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j8YACgkQMF6GPKJC 6IkoAA//RhhsVoP5akj/sAuiH/qHenJjl37mAJPk+NHSm4trz37SD5s4pPLz985L kplOh7szc47BHomPlF6Dpxl2zgfHSljm7LrqYYFaR2QzcA74vJFqRKGNeS5cnndm miedaX8hYm1eGHKVJ1DH+vVQ5zHevo5VbR+sbaVI3Ia7daebQ8obHfYLAZp+Y6fk T8NIkj7gM55LOr74ue8auJaYZ1MBHsdu+rxNZ4PF5Lf21bOAQO159FifidU85q6h 8qJqFJnQ0KUb3J/gprTl1RayZK5uA4/Bgl1H/3/wxuOGZ1GFTo96Kqe/fTHIWgmh Xckp/e1J0Q6EbfW89G0hylpRWJbs7PhyCM7NQz/wnYo/vaWegLSk6fHXmDzzYY/M 9PLtAr8VS7d64hxhcFBGTFvYTGfKdaNa3fqzQEOucf7lp5WIPlQaJn/8yBIGIbNk VYRCIuEIjjWipXqGcEGTQJgwbkFZVkuo9G4xo4+ggtK5T+5IP2Hjb56ary/prNTj RZfQRYT4oIov7SAf5VCGcz9ppJzjgIrWPLBbQkqsVcC8upGBtUuVwUJOcO12ctvV 9TeYEQxx/TXjdzORVPzax0Z72D8gXblO39cOULRe4xL4lyXKGubwXpsD7S/jGwc7 K2+0ze/f8adOPMGhctNzTl5sWmz7Mryrvd+uLMFEqv0U7nu4RTk= =+lHz -----END PGP SIGNATURE----- parent f58216efe8b13e487e95a058cd20d84690125755 author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762355 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j7MACgkQMF6GPKJC 6IlZohAAn+rsZSf2hDLEQnUlLfsH1V2XLhtqUCEwa3PaNhEpjMPX5VijvcR9aXvy Hd3FAeiyir7pQupMxf9+GjrZwFUzCp6U3GP8gp0kl1AfllTCQDzYhw6v/NeNOX3/ dowQaXpOccCib7TzKlpuqfvCnUcaxRWD9GHfoELmbrPMKVKXTyboIdHxLI9uWdVY 0lMgUCBJb2oRlQyq4XzWMwLBQYftockCmtpUqkC+AN+kXf2xm3ur9FEihGaLBmKg JlCtVM37PrT2zmeoMQKFxh+zsCyLUIN/4TlnKThu86GKLr5QPyBa7ePAdYX2WE45 6CaOdrjOI1U3vsg5q/xm+z+vZlLPQyhPc26ailGi5a1hGMHynCbxUlf1c1im9ciY vtsCwWW+Zjkp3Y/9tEA8E0xfxr9I/FfK/zNSxMCg7vcS5G1k2RXvGrIXp3aEgW09 zkxKxZFTEnaAoJwJTIXbn5M00ma6v5OYPupRwa4RhoTLPBkMQlXgfx63EdKIucsL rkJmIt/zjE6CWdomTiXK+cLU43U6ZvPsYgxD6BsieOrL2qHb9QNqZFTRg+8hY1RQ onUOfWK/5vUylutt42OFKn/WiEK3GfHSiIrhDr2Vhqi5Fj9aWODJHhKfJyrJqD2t l/uXVesblY+sEi5N9FztKOA8nJbENnfz1KQrNq14oCtST4dycvM= =/Lr+ -----END PGP SIGNATURE----- parent f58216efe8b13e487e95a058cd20d84690125755 author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762333 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j50ACgkQMF6GPKJC 6IlIRA/+OksmtDktKOs61tgkGuX4RK/G/BTTcjtKiQe/B9wiTJx1RoMvFcKIeuvm QCUgC45axhO+fSO8jc1+hsS8Ml2VxEjwAdW1iv09j/PNMJ8oOokVezi9RIpZbECp 5ITRVDEOC0dXxSDPv9w6rHM4aQVDtPYFHe3hZ65UnT1VLPChpnBb6pg6VaopiWW8 vslQjImlPQIvvpfsy3115ClQj4e2/IOCMo2Ms5Ey6i0Lu5UwV4aoOlTCTNZy5a+S 6FeOiV5lAqpR9bind2vHU9TyE8gzd++QEaQr8PpM5j3F9Cmym03YfwC84znU3Xuh UPJfBHzzmlzi7FM1Hl3vXEpBxD93xE8NZTXTfxhcTeQamwwwtcBpbUE/Xp1AllKw k1i7bnG+kqbCOqbh2sTmQASfX/XcBfCKLDDEDnX7Czd5rD2pjyJ1eziUffQLd50J qYriVxX6lSBgrkIsqsn9jDX44w+Ilj6Uj1tOJRKwaz+a9RBvvTtsghN0ojoOMCZ+ pB2Ag/Noa3ifRJqpIlGZKCGe7NYmTZPejU2lurzz9/5cvRh96CP/vjL6N7oIo84w mPuEuywjlXlboe65MfSFK8E3/H6QO6KirjE1ZeIoaSMEm8Dpgu4P6oe0WxVLqNG0 gjd9lXImIM8XVlbQU9RtNfw9hegsOvFyAjEAYfspLl6i/Pu7PEs= =iw7u -----END PGP SIGNATURE----- parent f58216efe8b13e487e95a058cd20d84690125755 author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762313 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j4kACgkQMF6GPKJC 6Im5OxAAkuCPtm0YtHjFHtcAvcCeT4+BFgQltKIvqT/zDOSLOkWvNSnPw41zeIpb KDqMZE1NzZTVV4bJ842dIjQ7L1sJ54ASCtp4uU9PHDsKzYifNucCkgWnL5i6Ra/G npislcslCjqGS/dLo3pu8YL+SN18TZOS6lrX6f8Io/eS4FqDOWUBhhArt+c6JNVc 91F9Pl+O4W0kGnk08x00gHKCtcHufHawh6WE8oFDXyq07VVQulOQnAQRW12XbELQ +gvN4AwnGEKuzNqmPR7qs0Mrj+TdDBdRwbVvhhrjDka20+wcVF4aAupabqekaOG4 2L8eaQwNXapwQagkAKX+k1p9Al5K06hbnItxTLKtJWyKyj7azADANhpMoVD9dh55 6+hBgCjbpB2cZDhEDKRbgSvEfKJW/N37F4F6jHFoc7cacGDSxa4eZchaGGFLhNM2 KhyRylvVA/myW3ddZPeEtTfn2BiB6uW8L1QlK6uAHnMDjnA+Nr9LN/CnDXuVlCBk +jcL84SX1ieglqJ0QOUMbLI4IMLX7AwNAA3777rXEu0P01SMQ9wdpGqwD0o0xXOJ TAV3aSGtPngFRnjwxu7PM2KFmZ2j2XLz86pVqk6AE/FUHOUKSRvIiTUGjoCr86Lu PAIkkoPj+tHd7q6tgIZLMVe2wbonuKAX3cqy/MEArJV5UtNSuV8= =/Bc9 -----END PGP SIGNATURE----- parent f58216efe8b13e487e95a058cd20d84690125755 author Vikrant Puppala <vikrant.puppala@databricks.com> 1726073284 +0530 committer Vikrant Puppala <vikrant.puppala@databricks.com> 1727762289 +0530 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEuLQK5O9DgM4B7E4mMF6GPKJC6IkFAmb7j3EACgkQMF6GPKJC 6Il7gQ//ccGZpelxwwUIdDntMrDdgEx3QY1HGLWzL8myQz19Wkv3bE/tiX4aapsw 7d9SnFRog5HD0h+oD10acTwPAQZcGkn+Fgx/3oAU6KobU7p6QcxXhyyLdI1gTdKq QflAO/tG+XHow/mTB8K/D0xS6NwI3SVOT8SA2ygUfv0n4rJYWDVLMTFWoyE+nDPC Ah1fkCLMYgdq3Oi5Nz2BH4ZScSKrfEIelhpIX6sVOsuVRoDXFKs5GXotih9GUK4U zwnj8c2nAaAq1Ub1SAj7Esns+wbj8FXNRZoAwOPlgyXtt7p1kil5P9nRTaJX2ic7 gJBcAteVk47971XLELFwkc8MlH6HHUUcBOiTNbWfvoHeB80MdCTKpVS6H9tDYLFf LUMGd7nEbk1wse95GRc2sh7NthHie54nZ2fspjNCQ6UXjEd+Qnc14D8R2bfxYjhR dxZExrpLgk0rj3cyrSWNKBAf+PihFOfYEWv7OiRkfS1udfWI4S4QEArYmKGzF/hy jWH4s94Gm22WpXgEQIjtrOAnfJBkW/Y7rQm9WnQaXUKwteKXv+uHstq4C8tww/0f S8JlJhTFqjjlJeLLqH/aY2KIFe53wK3cU0b6AgeOJ7muSZZevEdd4lCO5RgyhAqe do5DLrZIJIVicTAaCBH8LIE2NErBaUlhTkKxnbaBfptr/pcsK7s= =tQP5 -----END PGP SIGNATURE----- metastore api Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> regen Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> remove doc changes Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> [SPARK] gitignore signing keys and access tokens (#396) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Sister of PR #379 cc: @creechy Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> enable Delta tests and fix gs/abfs credential (#468) **Description of changes**  Use the Delta 3.2.1 RC so that these ignored Delta tests can be enabled. Also fixes the gs/abfs credential support in the Delta tests, by disabling the file system cache, the same as we did for s3. --------- Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Fix behavior of creation of managed tables (#449) **Description of changes**  UC managed tables should use the managed locations from the UC server. We should not leave it to the underlying `DeltaCatalog` which will generate the default table location using the Spark warehouse path. --------- Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Expose and Publish default port 8080 for start-uc-server-docker. (#440) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [X] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** By default, uc-server runs on ports 8080 (transcode) and 8081. Without --server argument, UC CLI/UI only connects to the transcode port 8080 by default. This change is to expose both port 8080/8081 in start-uc-server-docker so that * Users following documents (default: 8080) won't fail to connect the server running in docker. * CLI/UI won't need to specify --server argument. ``` derek@dev-linux unitycatalog (docker_expose) $ docker/bin/start-uc-server-docker ❤️ Starting unitycatalog:0.2.0-SNAPSHOT. 💡 Use Ctrl+C to stop and remove the container. ... derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list --server http://localhost:8081 ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list Exception in thread "main" java.lang.RuntimeException: io.unitycatalog.client.ApiException: java.net.ConnectException at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:165) Caused by: io.unitycatalog.client.ApiException: java.net.ConnectException ... ``` ``` derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ ``` --------- Co-authored-by: yaoderek <yaoderek@outlook.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> update docs by running `sbt generate` (#474) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  The API yaml file was updated but the docs weren't, just bringing them back in sync Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Feat/login UI (#450) **PR Checklist** - [x ] A description of the changes is added to the description of this PR. - [x ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Allows users to login via google auth once auth configuration is in place and users have been added to users table. <img width="1064" alt="Screenshot 2024-09-11 at 12 37 00 PM" src="https://github.com/user-attachments/assets/7f5db283-2604-4e6a-a179-f17d6c5537d1"> <img width="323" alt="Screenshot 2024-09-11 at 12 38 38 PM" src="https://github.com/user-attachments/assets/1824e68a-6a76-4f11-87f4-d49522a54a16"> Closes #430 Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Add owner, createdBy, updatedBy to all securables and regen openapi docs (#469) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Added owner, createdBy and updatedBy to all securable info objects and database tables Regenerated openAPI docs Updated the id names for model versions and registered models Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Run `javafmtAll` on main repo (#479) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Format the test file using javafmtAll Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Enable identity logging on create/update paths (#483) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Enable identity tracking for all create and update paths for securables in UC OSS Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Specify release settings for spark connector (#484) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Specify release settings for spark connector Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> [SPARK] Test parquet table creation by Spark 3.5.3 (#357) This PR is to test parquet table creation by create table or create table select as Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Build/release fixes (#489) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix a few issues in the 0.2.0 release process. - explicitly specify release settings on a few subprojects - fix some cross-building issues - move `controlApi` dependency to CLI project, it doesn't need to be published / marked as a spark dependency Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> UI READ support for models and versions (#460) **PR Checklist** - [x ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Users can browse models in the UI, view details of models, including versions. Version details page displays version data (created_by and updated_by coming soon with access control changes). <img width="1299" alt="Screenshot 2024-09-10 at 12 15 43 PM" src="https://github.com/user-attachments/assets/1591390b-d986-4ab3-8d2e-9c1bf66fcb97"> <img width="1392" alt="Screenshot 2024-09-10 at 12 16 02 PM" src="https://github.com/user-attachments/assets/2982ed79-91d2-49a9-8391-f31861e3a9fe"> <img width="1304" alt="Screenshot 2024-09-10 at 12 16 15 PM" src="https://github.com/user-attachments/assets/9dc8340e-8bc5-4e9f-80a9-242c3808e26e"> <img width="397" alt="Screenshot 2024-09-10 at 12 36 31 PM" src="https://github.com/user-attachments/assets/6c69a6a9-7864-479b-9078-06ab63c5f45b"> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Add temporary path credentials api (#478) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Added temporary path credentials openapi spec and implementation Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Fixed UC Cli NullPointerException for invalid entity types (#445) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Invalid entity type leads to Null pointer Exception in output. This change makes sure that entity type is validated first before further checks. (entering a invalid entity type into bin/uc) Before the fix: ``` derek@dev-linux unitycatalog (main) $ bin/uc sdfjio Please provide an operation. Please provide a valid sub-command for sdfjio. Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.util.Map.keySet()" because "entityOptions" is null at io.unitycatalog.cli.utils.CliUtils.printEntityHelp(CliUtils.java:525) at io.unitycatalog.cli.UnityCatalogCli.validateCommand(UnityCatalogCli.java:177) at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:105) ``` After the fix: ``` derek@dev-linux unitycatalog (fix_npe) $ bin/uc sdfjio Invalid entity provided: sdfjio Usage: bin/uc <entity> <operation> [options] Entities: schema, volume, model_version, auth, catalog, function, registered_model, user, table By default, the client will connect to UC running locally at http://localhost:8080 To connect to specific UC server, use --server https://<host>:<port> Currently, auth using bearer token is supported. Please specify the token via --auth_token <PAT Token> For detailed help on entity specific operations, use bin/uc <entity> --help ``` Co-authored-by: yaoderek <yaoderek@outlook.com> Co-authored-by: Denny Lee <denny.g.lee@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Update UI Readme (#459) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Removed the warning message. See [discussion](https://github.com/unitycatalog/unitycatalog/discussions/349#discussioncomment-10593858). - Add a title, an intro and an [image](https://github.com/unitycatalog/unitycatalog#interact-with-the-unity-catalog-ui). - As an idea, **I add a broken link** - [UI Documentation](https://github.com/unitycatalog/unitycatalog/tree/main/docs/ui). - Update the Get started section. - Add a References section to notate the merged. - Commented out Authenticate and Login section. Please let me know what you think.  --------- Signed-off-by: Juan Diaz <juanchodis@hotmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Mark spark-sql dependency as Provided (#490) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** We don't want the dependency resolvers to pull in an extra version of spark Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> add new acl fields to all entities (#491) **PR Checklist** - [ x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Add the `owner` `created_by` and `updated_by` fields to all other entities (models already has them from previous PR). If the fields are populated, they will display in the details sidebar of each entity. Also, removing non-functional "add/edit description" button from version details page (missed this straggler in previous PR). Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Added a python script explaining how to integration test mlflow with UC OSS (#448) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Added a script that acts as an integration test. Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Fix feature warning by adding import clause in UCSingleCatalog.scala (#427) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fixes #425 Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> docs: add docs for ORC tables (#388) This PR adds a documentation page that shows users how to work with ORC tables. Will close #206 --------- Signed-off-by: Avril Aysha <68642378+avriiil@users.noreply.github.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Docs: Managed vs External tables (#471) This PR adds a documentation page for managed vs external tables. Will close #221 Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> UC should work without Delta Lake (#467) **Description of changes**  If Delta Lake is not included in the class path, UC should still work and support file sources. --------- Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> docs: fix Daft integration guide (#498) **Description of changes** Updates the documentation around integration with Daft so that the guide can be followed without running into errors. - `daft.read_delta_lake` does not exist, the right API is `daft.read_deltalake` (see the related Daft docs [here](https://www.getdaft.io/projects/docs/en/latest/api_docs/doc_gen/io_functions/daft.read_deltalake.html#daft.read_deltalake)) - Daft must be installed including extras `unity` and `deltalake`, otherwise `daft.read_deltalake` will raise `ModuleNotFoundError` Signed-off-by: Péter Gyarmati <dev.petergy@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Added initial documentation for models (#447) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Initial version of model documentation. Quickstart manually verified on my local machine. --------- Co-authored-by: Matthew Powers <matthewkevinpowers@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> update from comments Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> fix: show detail message while create catalog (#497) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Name validation may fail when creating a new catalog, but it always shows a general error message which can be misleading. Add a detailed message to the error hint to help users fix their catalog name inputs. Before: ![image](https://github.com/user-attachments/assets/d82a8888-0094-4af1-aedf-a565aada890f) After: <img width="1229" alt="image" src="https://github.com/user-attachments/assets/4dc8bdbc-bec0-48a4-9134-035d68e54b87"> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Updated Delta 3.2.1 to new RC (#505) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** As the title says Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Basic server access control (#378) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** This feature builds upon the authentication functionality to add access controls to UC server endpoints. - Access control is optionally enabled through server server configuration (`server.authorization=enable`) - Resource endpoints all have access control rules that match access rules for Databricks Unity Catalog. - An access control is defined by a principal, a resource and a privilege. - Principals are currently only user entities. - Access controls are hierarchically applied - A privilege assigned to a resource for a principal is active for all the children. - A command-line interface is available to add, update, and delete authorizations. - The root of the hierarchy is a catalog. - There is a new concept of a metastore, which is primarily a resource to assign server level privileges with. The initial set of privileges defined are - `CREATE CATALOG` - allows the principal to create catalogs - `USE CATALOG` - allows the principal to access/use a catalog - `CREATE SCHEMA` - allows the principal to create schemas within a catalog - `USE SCHEMA` - allows the principal to access/use the schema and child tables - `CREATE_TABLE` - allows the principal to create tables in the schema - `SELECT` - allows the principal to run queries against table(s) - `CREATE_FUNCTION` - allows principal to create functions in the schema - `EXECUTE` - allows the principal to execute function(s) - `CREATE_VOLUME` - allows principal to create volumes in the schema - `READ VOLUME` - allows the principal to access volumes within the catalog - `CREATE MODEL` - allows the principal to create models within a schema The rules are generally more complex than summarized above, check Databricks documentation or the code for more details. The CLI interface provides a new `permission` command. Examples ``` bin/uc permission create --securable_type catalog \ --name mycatalog --principal user@myorg.com --privilege "CREATE CATALOG` bin/uc permission get --securable_type catalog --name unity binuc permission delete --securable_type table \ --name mycatalog.myschema.table --principal user@myorg.com --privilege SELECT ``` The framework attempts to separate the concern of access control from the rest of the request handling and processing through annotation-based configuration, via `@AuthorizeExpression`, `@AuthorizeKey` and `@AuthorizeKeys` added to each service entry point. All requests are routed through the `UnityAccessDecorator` which decodes the request, parameters and authorization configuration to evaluate whether access should be allowed or denied. --------- Co-authored-by: Vikrant Puppala <vikrant.puppala@databricks.com> Co-authored-by: Vikrant Puppala <vikrantpuppala@gmail.com> Co-authored-by: Xiang Xu <xiang.xu@databricks.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Fix duplicate index name. (#492) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Database index names apparently need to be unique. The same name `IDX_NAME` was used in multiple locations. Related Issue https://github.com/unitycatalog/unitycatalog/issues/493 Co-authored-by: Vikrant Puppala <vikrant.puppala@databricks.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Change wording on successful identity provider validation. (#509) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Simple wording change for CLI login process. Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> support create external delta table (#473) **Description of changes**  In `UCSingleCatalog#createTable`, we should get credentials by path and put the credential into the table properties, so that the underlying `DeltaCatalog` can access the Delta log when creating external delta tables. --------- Signed-off-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Include Rapidstart.md to run Spark, MLflow, and Auth integration end-to-end (#513) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Include the rapidstart.md which is an extension to the UC quickstart. It also includes instructions on how to utilize Spark, MLflow, and Auth integrations.  --------- Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Co-authored-by: Jamie Knight <jamie.knight@databricks.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Co-authored-by: Kris Concepcion <84737625+kriscon-db@users.noreply.github.com> Co-authored-by: Mocker <1179360+creechy@users.noreply.github.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> fix ui readme image link (#514) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Mlflow docs patch (#515) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Minor formatting pages to the docs/quickstart including the MLflow section.  Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Add access control to temporary credentials endpoints. (#511) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** This is a follow on to the overall access control feature to add access control to the credential vending endpoints. The access control rules for credential vending follows the same rules as getting the entity. This resolve Issue https://github.com/unitycatalog/unitycatalog/issues/501 Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Remove delta 3.2.1 prerelease resolver, official release is out (#518) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Remove PATH_REFRESH as it is an unused path operation (#520) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated Remove `PATH_REFRESH` as it is an unused path operation.  Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Update UC docs (#526) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed various formatting and configuration settings for the docs site. - Added Spark integration - Added Model integration - Includes coded tabs, admonitions, linked tabs, and other docs improvements.  --------- Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Co-authored-by: Kris Concepcion <84737625+kriscon-db@users.noreply.github.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Fix javadoc generation issue (#519) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix javadoc generation issue e.g. `unknown tag: AuthorizeKey` Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Include UI and auth docs (#528) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Including auth and UI docs. Slight restructuring of docs hierarchy. Clean up missing link, add cross linking, etc.  --------- Co-authored-by: Jamie Knight <jamie.knight@databricks.com> Co-authored-by: Shawn Chen <ychshawn@gmail.com> Co-authored-by: Mocker <1179360+creechy@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Feat/delete model version (#517) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Ability to delete a model version from the UI. <img width="425" alt="Screenshot 2024-09-26 at 2 29 57 PM" src="https://github.com/user-attachments/assets/fb27e047-f642-4221-8ef2-3b5ea96091a5"> <img width="522" alt="Screenshot 2024-09-26 at 2 29 51 PM" src="https://github.com/user-attachments/assets/d3b83207-3b21-44c5-8089-04d5370c3118"> <img width="226" alt="Screenshot 2024-09-26 at 2 29 41 PM" src="https://github.com/user-attachments/assets/b4726f0e-b5ad-4ac5-a1b6-77c9a3aba815"> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Update docs to include CLI managing users (#530) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed image loc in UI doc - Added managing users to CLI doc  Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Update auth.md (#531) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Clarification of Identity providers supported  Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> update from comments Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> Update docs by running `build/sbt generate` (#534) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Only run `build/sbt generate` to update the docs without any other changes Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com> [SPARK] gitignore signing keys and access tokens (#396) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Sister of PR #379 cc: @creechy enable Delta tests and fix gs/abfs credential (#468) **Description of changes**  Use the Delta 3.2.1 RC so that these ignored Delta tests can be enabled. Also fixes the gs/abfs credential support in the Delta tests, by disabling the file system cache, the same as we did for s3. --------- Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Expose and Publish default port 8080 for start-uc-server-docker. (#440) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [X] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** By default, uc-server runs on ports 8080 (transcode) and 8081. Without --server argument, UC CLI/UI only connects to the transcode port 8080 by default. This change is to expose both port 8080/8081 in start-uc-server-docker so that * Users following documents (default: 8080) won't fail to connect the server running in docker. * CLI/UI won't need to specify --server argument. ``` derek@dev-linux unitycatalog (docker_expose) $ docker/bin/start-uc-server-docker ❤️ Starting unitycatalog:0.2.0-SNAPSHOT. 💡 Use Ctrl+C to stop and remove the container. ... derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list --server http://localhost:8081 ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list Exception in thread "main" java.lang.RuntimeException: io.unitycatalog.client.ApiException: java.net.ConnectException at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:165) Caused by: io.unitycatalog.client.ApiException: java.net.ConnectException ... ``` ``` derek@dev-linux unitycatalog (docker_expose) $ ./bin/uc catalog list ┌─────┬────────────┬──────────┬─────────────┬──────────┬────────────────────────────────────┐ │NAME │ COMMENT │PROPERTIES│ CREATED_AT │UPDATED_AT│ ID │ ├─────┼────────────┼──────────┼─────────────┼──────────┼────────────────────────────────────┤ │unity│Main catalog│{} │1721241605334│null │f029b870-9468-4f10-badc-630b41e5690d│ └─────┴────────────┴──────────┴─────────────┴──────────┴────────────────────────────────────┘ ``` --------- Co-authored-by: yaoderek <yaoderek@outlook.com> update docs by running `sbt generate` (#474) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  The API yaml file was updated but the docs weren't, just bringing them back in sync Run `javafmtAll` on main repo (#479) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Format the test file using javafmtAll [SPARK] Test parquet table creation by Spark 3.5.3 (#357) This PR is to test parquet table creation by create table or create table select as Build/release fixes (#489) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix a few issues in the 0.2.0 release process. - explicitly specify release settings on a few subprojects - fix some cross-building issues - move `controlApi` dependency to CLI project, it doesn't need to be published / marked as a spark dependency Add temporary path credentials api (#478) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Added temporary path credentials openapi spec and implementation Fixed UC Cli NullPointerException for invalid entity types (#445) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Invalid entity type leads to Null pointer Exception in output. This change makes sure that entity type is validated first before further checks. (entering a invalid entity type into bin/uc) Before the fix: ``` derek@dev-linux unitycatalog (main) $ bin/uc sdfjio Please provide an operation. Please provide a valid sub-command for sdfjio. Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.util.Map.keySet()" because "entityOptions" is null at io.unitycatalog.cli.utils.CliUtils.printEntityHelp(CliUtils.java:525) at io.unitycatalog.cli.UnityCatalogCli.validateCommand(UnityCatalogCli.java:177) at io.unitycatalog.cli.UnityCatalogCli.main(UnityCatalogCli.java:105) ``` After the fix: ``` derek@dev-linux unitycatalog (fix_npe) $ bin/uc sdfjio Invalid entity provided: sdfjio Usage: bin/uc <entity> <operation> [options] Entities: schema, volume, model_version, auth, catalog, function, registered_model, user, table By default, the client will connect to UC running locally at http://localhost:8080 To connect to specific UC server, use --server https://<host>:<port> Currently, auth using bearer token is supported. Please specify the token via --auth_token <PAT Token> For detailed help on entity specific operations, use bin/uc <entity> --help ``` Co-authored-by: yaoderek <yaoderek@outlook.com> Co-authored-by: Denny Lee <denny.g.lee@gmail.com> Mark spark-sql dependency as Provided (#490) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** We don't want the dependency resolvers to pull in an extra version of spark add new acl fields to all entities (#491) **PR Checklist** - [ x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Add the `owner` `created_by` and `updated_by` fields to all other entities (models already has them from previous PR). If the fields are populated, they will display in the details sidebar of each entity. Also, removing non-functional "add/edit description" button from version details page (missed this straggler in previous PR). Added a python script explaining how to integration test mlflow with UC OSS (#448) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Added a script that acts as an integration test. Fix feature warning by adding import clause in UCSingleCatalog.scala (#427) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [x] If there is a related issue, make sure it is linked to this PR. - [x] If you've fixed a bug or added code that should be tested, add tests! - [x] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fixes #425 Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> docs: add docs for ORC tables (#388) This PR adds a documentation page that shows users how to work with ORC tables. Will close #206 --------- Signed-off-by: Avril Aysha <68642378+avriiil@users.noreply.github.com> Docs: Managed vs External tables (#471) This PR adds a documentation page for managed vs external tables. Will close #221 docs: fix Daft integration guide (#498) **Description of changes** Updates the documentation around integration with Daft so that the guide can be followed without running into errors. - `daft.read_delta_lake` does not exist, the right API is `daft.read_deltalake` (see the related Daft docs [here](https://www.getdaft.io/projects/docs/en/latest/api_docs/doc_gen/io_functions/daft.read_deltalake.html#daft.read_deltalake)) - Daft must be installed including extras `unity` and `deltalake`, otherwise `daft.read_deltalake` will raise `ModuleNotFoundError` Signed-off-by: Péter Gyarmati <dev.petergy@gmail.com> Added initial documentation for models (#447) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Initial version of model documentation. Quickstart manually verified on my local machine. --------- Co-authored-by: Matthew Powers <matthewkevinpowers@gmail.com> fix: show detail message while create catalog (#497) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Name validation may fail when creating a new catalog, but it always shows a general error message which can be misleading. Add a detailed message to the error hint to help users fix their catalog name inputs. Before: ![image](https://github.com/user-attachments/assets/d82a8888-0094-4af1-aedf-a565aada890f) After: <img width="1229" alt="image" src="https://github.com/user-attachments/assets/4dc8bdbc-bec0-48a4-9134-035d68e54b87"> Updated Delta 3.2.1 to new RC (#505) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** As the title says Fix duplicate index name. (#492) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Database index names apparently need to be unique. The same name `IDX_NAME` was used in multiple locations. Related Issue https://github.com/unitycatalog/unitycatalog/issues/493 Co-authored-by: Vikrant Puppala <vikrant.puppala@databricks.com> Change wording on successful identity provider validation. (#509) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Simple wording change for CLI login process. support create external delta table (#473) **Description of changes**  In `UCSingleCatalog#createTable`, we should get credentials by path and put the credential into the table properties, so that the underlying `DeltaCatalog` can access the Delta log when creating external delta tables. --------- Signed-off-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <cloud0fan@gmail.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Include Rapidstart.md to run Spark, MLflow, and Auth integration end-to-end (#513) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Include the rapidstart.md which is an extension to the UC quickstart. It also includes instructions on how to utilize Spark, MLflow, and Auth integrations.  --------- Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Co-authored-by: Jamie Knight <jamie.knight@databricks.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Co-authored-by: Kris Concepcion <84737625+kriscon-db@users.noreply.github.com> Co-authored-by: Mocker <1179360+creechy@users.noreply.github.com> fix ui readme image link (#514) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Add access control to temporary credentials endpoints. (#511) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** This is a follow on to the overall access control feature to add access control to the credential vending endpoints. The access control rules for credential vending follows the same rules as getting the entity. This resolve Issue https://github.com/unitycatalog/unitycatalog/issues/501 Remove delta 3.2.1 prerelease resolver, official release is out (#518) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**  Remove PATH_REFRESH as it is an unused path operation (#520) **PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated Remove `PATH_REFRESH` as it is an unused path operation.  Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Update UC docs (#526) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed various formatting and configuration settings for the docs site. - Added Spark integration - Added Model integration - Includes coded tabs, admonitions, linked tabs, and other docs improvements.  --------- Signed-off-by: Denny Lee <denny.g.lee@gmail.com> Co-authored-by: Kris Concepcion <84737625+kriscon-db@users.noreply.github.com> Co-authored-by: Bijan Houle <bijanhoule@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Fix javadoc generation issue (#519) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix javadoc generation issue e.g. `unknown tag: AuthorizeKey` Include UI and auth docs (#528) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Including auth and UI docs. Slight restructuring of docs hierarchy. Clean up missing link, add cross linking, etc.  --------- Co-authored-by: Jamie Knight <jamie.knight@databricks.com> Co-authored-by: Shawn Chen <ychshawn@gmail.com> Co-authored-by: Mocker <1179360+creechy@users.noreply.github.com> Co-authored-by: Tathagata Das <tathagata.das1565@gmail.com> Feat/delete model version (#517) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Ability to delete a model version from the UI. <img width="425" alt="Screenshot 2024-09-26 at 2 29 57 PM" src="https://github.com/user-attachments/assets/fb27e047-f642-4221-8ef2-3b5ea96091a5"> <img width="522" alt="Screenshot 2024-09-26 at 2 29 51 PM" src="https://github.com/user-attachments/assets/d3b83207-3b21-44c5-8089-04d5370c3118"> <img width="226" alt="Screenshot 2024-09-26 at 2 29 41 PM" src="https://github.com/user-attachments/assets/b4726f0e-b5ad-4ac5-a1b6-77c9a3aba815"> Update docs to include CLI managing users (#530) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** - Fixed image loc in UI doc - Added managing users to CLI doc  Update auth.md (#531) **PR Checklist** - [X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Clarification of Identity providers supported  Signed-off-by: Denny Lee <denny.g.lee@gmail.com>

# Conflicts: # README.md # api/README.md # api/all.yaml # docs/quickstart.md

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

**PR Checklist** - [ X] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Users can delete models from the UI. <img width="427" alt="Screenshot 2024-09-30 at 12 33 30 PM" src="https://github.com/user-attachments/assets/1729869b-8260-4dcb-b290-55c223f862ac"> <img width="522" alt="Screenshot 2024-09-30 at 12 33 24 PM" src="https://github.com/user-attachments/assets/87238db6-c5c7-42d1-99a1-90c8b323e248"> <img width="519" alt="Screenshot 2024-09-30 at 12 32 48 PM" src="https://github.com/user-attachments/assets/cbde51b3-b82f-4527-8105-926cd283b1bf"> <img width="180" alt="Screenshot 2024-09-30 at 12 32 37 PM" src="https://github.com/user-attachments/assets/2a506f2c-f396-473a-bded-a5720b06698b">

…les (unitycatalog#544) **PR Checklist** - [x] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes** Fix a bug where the path variables for UpdateModelVersion and UpdateRegisteredModel were being ignored in favor of the incorrectly added variables in the request object. - Request object has been fixed - Path variables are passed in correctly for both handlers and used - All existing code that relied on extraneous fields in generated code has been fixed - All tests continue to pass

**PR Checklist** - [ ] A description of the changes is added to the description of this PR. - [ ] If there is a related issue, make sure it is linked to this PR. - [ ] If you've fixed a bug or added code that should be tested, add tests! - [ ] If you've added or modified a feature, documentation in `docs` is updated **Description of changes**

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Co-authored-by: Ramesh Chandra <ramesh.chandra@gmail.com> Signed-off-by: Vikrant Puppala <vikrantpuppala@gmail.com>

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

vksx and others added 30 commits September 11, 2024 11:42

[SPARK] Test parquet table creation by Spark 3.5.3 (unitycatalog#357)

6ab4a64

This PR is to test parquet table creation by create table or create table select as

docs: add docs for ORC tables (unitycatalog#388)

4e5d6a5

This PR adds a documentation page that shows users how to work with ORC tables. Will close unitycatalog#206 --------- Signed-off-by: Avril Aysha <68642378+avriiil@users.noreply.github.com>

Docs: Managed vs External tables (unitycatalog#471)

c70d837

This PR adds a documentation page for managed vs external tables. Will close unitycatalog#221

dennyglee and others added 11 commits September 26, 2024 13:01

vikrantpuppala force-pushed the metastore-api branch 2 times, most recently from 8b4fe49 to 4c5e15e Compare October 1, 2024 05:52

vikrantpuppala force-pushed the metastore-api branch from 6b4afc8 to f9e2d3c Compare October 1, 2024 06:00

vikrantpuppala and others added 15 commits October 1, 2024 11:32

Merge remote-tracking branch 'unitycatalog/main' into metastore-api

a12f45e

# Conflicts: # README.md # api/README.md # api/all.yaml # docs/quickstart.md

revert unknown changes

f81126b

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Merge remote-tracking branch 'unitycatalog/main' into metastore-api

274cd30

clean

549e9ba

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

update description

d735df8

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

fix

966b93a

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

only keep metastore id in resp

79e68e5

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

update docs

111b3fd

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Update api/all.yaml

3b025cd

Co-authored-by: Ramesh Chandra <ramesh.chandra@gmail.com> Signed-off-by: Vikrant Puppala <vikrantpuppala@gmail.com>

Update api/all.yaml

2154fb0

Co-authored-by: Ramesh Chandra <ramesh.chandra@gmail.com> Signed-off-by: Vikrant Puppala <vikrantpuppala@gmail.com>

Update api/all.yaml

041c2ce

Co-authored-by: Ramesh Chandra <ramesh.chandra@gmail.com> Signed-off-by: Vikrant Puppala <vikrantpuppala@gmail.com>

update docs

6b354d9

Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add metastore API spec #7

Add metastore API spec #7

vikrantpuppala commented Sep 12, 2024

Add metastore API spec #7

Are you sure you want to change the base?

Add metastore API spec #7

Conversation

vikrantpuppala commented Sep 12, 2024